PORTRAIT IMAGE PROCESSING METHOD AND APPARATUS 



BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates io a portrait image processing method and a 
5 portrait image processing apparatus, and in particular, to a technique for extracting a 

portrait image from an original image and compositing the extracted portrait image and a 
background image which is prepared in advance. 

Description of the Related Art 

Conventionally, as a method of extracting a portrait image from an original 
10 image, there is known a method of photographing a person with a blue screen as a 

background (blue back) to acquire an original image and making use of a difference of 

colors (chroma) to extract a portrait image from the original image and insert the portrait 

image in another image (chroma key). 

In addition, Japanese Patent Application Publication No. 10-13799 discloses a 
15 technique for dividing a photographed image into a plurality of blocks arranged as a 

matrix and sorting the blocks into a background block with small movement and an 

object block with large movement (block including a portrait image) for each block 

according to a magnitude of movement among frames. 

However, there is a problem in that, although it is easy to extract a portrait 
20 image in the case of an original image which is photographed in a simple background 

such as the blue back, a portrait image cannot be extracted from an original image, which 

is photographed in an intricate background, satisfactorily. 

On the other hand, a method of sorting a portrait image from a background 

image disclosed in Japanese Patent Application Publication No. 10-13799 has a problem 
25 in that, since a magnitude of movement among frames for each block of a photographed 

image is utilized, the portrait image (a block including a person) and the background 

image cannot be sorted in the case of a still image or in the case in which the person does 

not move. 
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SUMMARY OF THE INVENTION 
The present invention has been achieved in view of such circumstances, and it is 
an object of the present invention to provide a portrait image processing method and a 
portrait image processing apparatus which, even in the case in which a portrait image is 
5 not extracted accurately from an original image, can form an image, which is obtained by 
compositing the extracted portrait image and the background image, as a natural image. 

In order to attain the above-described object, a portrait image processing method 
according to a first aspect of the present invention is characterized by comprising the 
steps of: extracting a portrait image from an original image including a person and a 
10 background; compositing the extracted portrait image and a background image prepared 
in advance to create a composite image; detecting a boundary of the person and the 
background from the original image; judging whether or not the detected boundary is a 
true contour of the person for each part of the boundary; and applying correction 
processing for concealing a boundary part, which is judged not to be a true contour of the 
15 person, to the boundary of the person and the background in the created composite 
image. 

Even in the case in which the portrait image is not extracted accurately from the 
original image (i.e., the boundary of the person and the background detected from the 
original image does not coincide with the true contour), all parts of the boundary are not 

20 necessarily inaccurate but include boundary parts with high certainty as a contour of a 
person and boundary parts with low certainty as a contour of a person. In the first 
aspect, in compositing the extracted portrait image and the background image, the image 
processing for concealing a boundary part is applied to the boundary parts with low 
certainty as a contour of a person such that a natural composite image is obtained. 

25 In the portrait image processing method of the first aspect, the correction 

processing may be image processing for overwriting another image on the boundary part 
which is judged not to be the true contour of the person. 

In the portrait image processing method of the first aspect, the correction 
processing may be image processing for shifting the portrait image such that the 

30 boundary part, which is judged not to be the true contour of the person, is outside a frame 
of the composite image. 
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A portrait image processing apparatus according to a second aspect of the 
present invention is characterized by comprising: a portrait image extracting device 
which extracts a portrait image from an original image including a person and a 
background; a background image recording device which stores a background image to 
5 be a background of a portrait image; an image compositing device which composites the 
extracted portrait image and the background image read out from the background image 
recording device to create a composite image; a boundary detecting device which detects 
a boundary of the person and the background from the original image; a judging device 
which judges whether or not the detected boundary is a true contour of the person for 
10 each part of the boundary; and an image correcting device which applies correction 

processing for concealing a boundary part, which is judged not to be a true contour of the 
person, to the boundary of the person and the background in the created composite 
image. 

In the portrait image processing apparatus of the second aspect, the image 
15 correcting device may perform image processing for overwriting another image on the 
boundary part which is judged not to be the true contour of the person. 

In the portrait image processing apparatus of the. second aspect, the image 
correcting device may perform image processing for shifting the portrait image such that 
the boundary part, which is judged not to be the true contour of the person, is outside a 
20 frame of the composite image. 

As described above, according to the present invention, in compositing a portrait 
image extracted from an original image and an appropriate background image, even in 
the case in which the original image includes an intricate background image and the 
portrait image is not extracted accurately from the original image (i.e., a boundary of a 
25 person and a background detected from the original image does not coincide with a true 
contour of the person), since the image processing for concealing a boundary part is 
applied to the boundary parts with low certainty as a contour of a person, a natural 
composite image can be obtained. 

BRIEF DESCRIPTION OF THE DRAWINGS 
30 Fig. 1 is a functional block diagram showing a main part of a portrait image 

processing apparatus in accordance with the present invention; 
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Figs. 2 A to 2D are diagrams used for explaining an embodiment of a portrait 
image processing method in accordance with the present invention; 

Fig. 3 is a diagram showing boundary parts with high certainty as a contour of a 
person and boundary, parts with low certainty as a contour of a person in a boundary of a 
5 person area and a background area of an original image; 

Figs. 4A and 4B are diagrams showing another embodiment of correction 
processing in the portrait image processing method in accordance with the present 
invention; and 

Fig. 5 is a diagram of a network system to which the portrait image processing 
10 method in accordance with the present invention is applied. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
A preferred embodiment of a portrait image processing method and a portrait 
image processing apparatus in accordance with the present invention will be hereinafter 
described in detail with reference to the accompanying drawings. 

15 Fig 1 is a functional block diagram showing a main part of the portrait image 

processing apparatus in accordance with the present invention. 

As shown in the figure, a portrait image processing apparatus 10 can be 
constituted by a personal computer or the like, and includes an image data input unit 12, 
a person area extraction unit 14, a compositing processing unit 16, a background image 

20 storage unit 18, a correction processing unit 20, a correction image storage unit 22, and 
an image data output unit 24. 

Original image data photographed by a digital still camera (hereinafter referred 
to as DSC) or the like is inputted to the portrait image processing apparatus 10 via the 
image data input unit 12. Note that this original image data is portrait image data which 

25 is obtained by photographing a person in an arbitrary background as shown in Fig. 2A. 
As the image data input unit 12, a USB interface, an infrared ray communication (IrDA) 
interface, an Ethernet interface, and a wireless communication interface may be used 
other than a media interface for DSC media. A user chooses appropriately, according to 
a medium in which the original image data is stored or a recording format of the original 

30 image data, which interface is applied to the image data input unit 12. 
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The original image data inputted to the portrait image processing apparatus 10 is 
outputted to the person area extraction unit 14, where a person area is distinguished from 
a background area and extracted (see Fig. 2B). 

As a method of extracting the person area from the original image, for example, 
5 characteristics extracting processing is performed for extracting facial parts such as eyes, 
a nose, a mouth in the original image. In the characteristics extracting processing, 
wavelet transformation is performed with respect to the original image to extract and 
quantize wavelet coefficients of appropriate positions and frequencies. The wavelet 
coefficients are subjected to matching processing to be matched with facial parts 
10 dictionary data which is prepared by performing the same characteristics extracting 
processing for a large number of sample images in advance, and facial parts are 
extracted. 

A part from which the facial parts are extracted is judged to be a face, to which 
division processing according to colors and textures is applied. This is processing for 

15 cotlectively dividing areas of similar colors and textures, and for example, a skin color 
area including coordinates of eyes is set as a face area and an area of black or brown 
slightly above the coordinates of eyes is set as a hair area to extract a person area. In 
addition, matching processing for matching person area dictionary data, which indicates 
an average positional relation between a position of eyes and a boundary of a person and 

20 a background, and a boundary of a person area and a background area obtained from the 
original image is performed to find a boundary of a person area and a background area. 

In addition, there are other methods such as a method of applying filter 
processing for extracting a boundary of a person and a background from a 
high-frequency component in an original image to extract a person area from the original 

25 image, a method of extracting a skin color in an original image, sequentially applying 
area extension to connected areas, which seem to belong to an identical area, from a 
certain point of the skin color area, extracting a face area depending upon whether or not 
an area extracted in this way is a shape of a face, and extracting a hair area above the 
face area, a neck and chest area below the face area, and the like, to thereby extract a 

30 person area. Note that there are various methods are possible as a method of extracting 
a person area from an original image, which is not limited to the above-described 
methods. 
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Image data of the person area extracted by the person area extraction unit 14 is 
outputted io the compositing processing unit 16. The compositing processing unit 16. 
composites portrait image data, which is inputted from the person area extraction unit 14, 
and background image data, which is read out from the background image storage unit 
5. 18, and outputs composite image data of the portrait image data and the background 
image data to the correction processing unit 20. 

Fig. 2C shows a composite image obtained by compositing a portrait image with 
a background image. Note that, as the background image, a desired background image 
to be composited with the portrait image may be selected out of a plurality of background 
10 images stored in the background image storage unit 18 in advance or a background 

image to be composited with the portrait image may be inputted separately. A method 
of acquiring the background image is not limited to this embodiment. 

First, the correction processing unit 20 determines whether a boundary part is a 
boundary part with high certainty as a contour of a person or a boundary part with low 
15 certainty as a contour of a person from the boundary of the person area and the 

background area, which are used for the extraction of the person area in the person area 
extraction unit 14, or a boundary detected from an external circumference of the 
extracted person area. 

Fig. 3 is a diagram showing boundary parts with high certainty as a contour of a 
20 person and boundary parts with low certainty as a contour of a person in a boundary of a 
person area and a background area. Encircled boundary parts indicate parts which are 
determined to have low certainty as a contour of a person. 

The boundary parts with low certainty correspond to, for example, a boundary 
part where a length between coordinate points on the boundary is partially larger than a 
25 decided value due to unevenness of the boundary, a boundary part which is out of a range 
of a reference contour line (reference contour line of a person including a head, a neck, 
shoulders, etc.), which is collected from contours of a large number of people, added 
with a predetermined margin, or a boundary part which has a shape of each part of the 
boundary largely different from a shape of the reference contour. 
30 Note that, in the case in which a person is photographed with a screen or the like 

having a single color and a uniform density as a background, a boundary of the person 
and the background detected from an original image coincides with a true contour of the 
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person in most cases. However, in the case in which a person is photographed in a 
background with an intricate pattern or color, a boundary of the person and the 
background cannot be detected accurately from an original image, and the boundary of 
the person and the background includes parts which do not coincide with a true contour 
5 of the person. 

When the boundary part with low certainty as a contour of a person is detected 
as described above, the correction processing Unit 20 performs image processing for 
concealing the boundary part. In other words, the correction processing unit 20 reads 
out an appropriate correction image from the correction image storage unit 22 and 
10 overwrites this correction image on the boundary part with low certainty as a contour of 
a person. 

Fig. 2D shows a composite image after correction in which the correction image 
is overwritten on the boundary part with low certainty by the correction processing unit 
20. in the embodiment shown in Fig. 2D, an image of leaves is used as a correction 

15 image. However, it is preferable to select ah appropriate correction image, which does 
not have a sense of incongruity with respect to the background image, from the 
correction image storage unit 22. In addition, the overwriting processing may be 
performed with a hat as a correction image in the case in which the boundary part with 
low certainty is in the parietal area, or a shawl as a correction image in the case in which 

20 the boundary part with low certainty is on the shoulders. 

The composite image data corrected by the correction processing unit 20 is 
outputted as image data, in which the background is composited, via the image data 
output unit 24. Examples of a form of outputting the image data include a form of 
outputting the image data to a monitoring device or a printer, and a form of recording the 

25 image data in a file format in an external recording medium such as a PC card or a 

CD-ROM or a hard disk incorporated in the portrait image processing apparatus 10. In 
addition, a form of transferring the image data to other apparatuses via communication 
means is also possible. Communication may be performed by wire or by radio. 

Figs. 4 A and 4B are diagrams showing another embodiment of the correction 

30 processing by the correction processing unit 20. 

Fig. 4A is a diagram showing boundary parts with high certainty as a contour of 
a person and boundary parts with low certainty as a contour of a person in a boundary of 
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a person area and a background area. Encircled boundary parts indicate parts which are 
determined to have low certainty. As shown in the figure, the boundary parts with low 
certainty concentrate in a left part A of the boundary. 

In the case in which the boundary parts with low certainty concentrate only in a 
5 part of the boundary of the person area and the background area as shown in Fig. 4 A, the 
correction processing unit 20 shifts a portrait image such that the boundary part with low 
certainty (left part Al) is out of a composite image. 

Note that the correction processing unit 20 may be adapted to perform correction 
processing which is a combination of the overwriting processing according to the 
10 correction image shown in Fig. 2D and the shift of the portrait image shown in Fig. 4B. 

The portrait image processing apparatus 10 can be realized by a personal 
computer. However, the portrait image processing apparatus 10 is not limited to this 
but may be realized by a service server or the like for image processing on a network. 

Fig. 5 is a diagram of a network system to which the portrait image processing 
15 method in accordance with the present invention is applied. 

In Fig. 5, reference numeral 30 designates a cellular phone with camera which is 
connectable to a network 40 such as the Internet, and 50 designates a computer (PC) of a 
user which is connectable to the network 40. Note that a DSC 52 is connected to the PC 
50 via an interface such as a USB such that the PC 50 can capture an image from the 
20 DSC 52. 

In addition, a service server 60, which performs the same image processing as 
the image processing in the portrait image processing apparatus 10, a print server 70 
which prints to output a composite image processed in the service server 60, or the like 
are connected to the network 40. 

25 In the case in which a user uses a compositing processing service of a 

background image, which is provided by the service server 60, with the cellular phone 
with camera 30 or the PC 50, the user accesses a web site of the service server 60 to 
upload an image requesting compositing processing of the background image to the 
service server 60. In addition, the service server 60 can present a list of background 

30 images or the like to the user and cause the user to select a background image. 

The service server 60 includes a server computer 62, which has the same 
function as the portrait image processing apparatus 10 shown in Fig. I and a 
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communication function, and a large capacity storage 64, which stores the image 
uploaded from the user and user information such as a user ID and a mail address. 
Upon receiving a request for compositing processing for an original image uploaded 
from the user with a background image, the service server 60 extracts a portrait image 
5 from the original image, performs compositing processing of this portrait image and a 
background image selected in advance, and applies correction processing to a boundary 
part with low certainty as a contour of a person in a boundary of a person area and a 
background area. Then, the service server 60 attaches an image, in which the 
background image is composited, created in this way to an e-mail and distributes the 

10 image to the cellular phone with camera 30 and the PC 50 of the user or distributes a 

mail attached with a URL for image download to the cellular phone with camera 30 and 
the PC 50 of the user. 

In addition, when an order for printing a composite image is received from the 
user, the service server 60 transfers the composite image to a print server 70. The print 

15 server 70 includes a server computer 72 and a print apparatus 74, and prints to output a 
composite image, in which a background is composited, with the print apparatus 74 on 
the basis of the composite image received from the service server 60. Note that a 
photograph print, which is printed and outputted by the print apparatus 74, is delivered to 
a destination such as a convenience store or a photo processing shop designated by the 

20 user or directly delivered to a home of the user. 
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